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identified  to  be  Pullup  (GCAS  warning)  ,  TF-Low  (Set  Cleaur*ance  Plane  [SCP]  violation 
warning) ,  Alpha-Side-Slip  (AOA-SS  caution) ,  and  Autopilot  Failed  (autopiot  failure 
caution) ;  (2)  The  prioritization  scheme  should  have  the  warnings  and  cautions  in  the 
following  order:  GCAS  warning,  followed  by  an  SCP  viclation  warning,  an  AOA-SS  caution, 
and  finally  an  autopilot  failure  caution;  finally,  (3)  Warning  messages  should  be 
presented  continuously  until  a  manual  response  is  performed,  w*\ile  caution  messages  should 
only  be  repeated  two  or  three  times.  A  portion  of  the  results  vort.-  /.ot  necessarily 
confined  to  the  F/FB/EF-111  cockpit  and  may  therefore  be  used  to  generalize  to  other 
cockpits.  The  followiing  is  a  description  of  the  more  general izable  results:  (1)  A  voice 
warning  mode  is  preferred  over  either  tone  or  light  alone;  (2)  a  tone  stall  warning  mode 
was  most  often  selected  by  the  subjects  over  voice  stall  warning  mode;  (3)  in  case  a  voice 
stall  warning  mode  is  to  be  considered,  then  most  appropriate  message  would  be  ’STALL'; 

€md  (4)  when  presenting  a  voice  message,  more  pilots  prefer  to  hear  the  voice  of  a  female 
ais  opposed  to  a  male  speadcer. 
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IllTBODnCTION 


An  effort  to  upgrade  the  F/FB/EF-111  aircraft,  through 
the  addition  of  a  Digital  Flight  Control  Computer  (DFCC) ,  is 
currently  underway.  One  of  the  new  safety  of  flight 
inclusions  into  the  Digital  Flight  Control  System  (DFCS)  is  a 
Voice  Message  Unit  (VMU)  .  The  VMU  will  be  used  as  a  means 
for  alerting  crew  members  to  flight  critical  conditions.  The 
following  are  four  alerts,  included  in  the  F-111  VMU,  which 
are  of  interest  to  this  study;  (1)  A  ground  collision  warning 
(which  uses  a  Ground  Collision  Avoidance  System  (GCAS) 
algorithm);  (2)  a  Set  Clearance  Plane  (SCP)  violation 
warning:  (3)  an  Angle  Of  Attack/Side  slip  (AOA/SS)  caution 
indicating  performance  degradation  in  Angle  Of  Attack  limiter 
and  adverse  yaw  compensation;  (4)  and  a  caution  informing  the 
crew  of  an  autopilot  malfunction. 

The  Crew  Station  Design  Facility  was  tasked  by  the  F-111 
System  Program  Office  (SPO)  to  evaluate  the  design  of  the 
Crew/VMU  interface  proposed  by  General  Dynamics. 

Current  military  standards  do  not  provide  specific 
directions  in  the  design  of  crew  members/VMU  interfaces. 

While  MIL-STD-1472C  (HUMAN  ENGINEERING  DESIGN  CRITERIA  FOR 
MILITARY  SYSTEMS,  EQUIPMENT  AND  FACILITIES)  contains  a 
section  specifying  the  design  of  a  general  purpose  audio 
display  interface,  the  information  is  incomplete  for 
applications  related  to  aircraft  cockpit  environments.  In 
turn,  MIL-STD- 1472C  section  5.3. 1.6  references  MIL-STD-411D 
(AIRCREW  STATION  WARNING  SIGNALS)  as  the  appropriate  document 
for  designing  audio  signals  for  aircrew  stations.  The 
following  is  the  verbal  auditory  warning  signals  excerpt  from 
MIL-STD-41 ID: 
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5.2.5  Verbal  auditory  warning  signals  -  Verbal 
warning  signals  shall  be  audible  signals  in  verbal 
form  indicating  the  existence  of  a  hazardous  or 
imminent  catastrophic  condition  requiring 
immediate  action  and  shall  only  be  used  to 
complement  red  warning  or  other  critical  visual 
signals.  The  verbal  warning  signals  shall  be 
presented  at  levels  which  will  insure  operator 
reception  under  noise  conditions  in  the  specific 
aircraft.  There  shall  be  provision  for  overriding 
and  reseting  the  signals.  The  signal,  when 
activated,  shall  always  start  at  the  beginning  of 
the  message  and  shall  continue  to  be  presented 
until  e i ther  : 

(a)  The  causative  condition  is  corrected. 

(b)  A  warning  of  higher  priority  is  presented. 

(c)  The  signal  is  silenced  by  manual 
actuation  of  the  override  switch. 

The  structure  for  verbal  warning  shall  be: 

(d)  General  heading  -  i.e. ,  the  system  or 
service  is  involved. 

(e)  Specific  subsystem  or  location 

(f)  Mature  of  emergency 

Example:  ENGINE  No.  1  HOT 

(d)  (e)  (f) 

Several  topics  remain  undefined.  These  include  such 
items  as  the  format  of  caution  verbal  messages;  inter-message 
time;  prioritization;  distinction  between  verbal  caution  and 
warning  messages;  etc. . 

The  following  write-up  describes  a  two  phase  study  which 
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attempts  to  resolve  some  general  issues  concerning  voice 
message  units,  as  well  as  issues  specific  to  the  selection 
of  the  F/FB/EF-111  voice  messages.  Phase  I  of  this  study 
involved  the  collection  of  questionnaire  data  from 
operational  F/FB/EF-111  crew  members.  The  questionnaire 
results  were  used  to  influence  the  design  of  a  Phase  II 
study,  in  which  pilots  were  given  the  opportunity  to  fly  a 
VMU  mechanized  FB-111  simulator  and  evaluate  its  overall 
adequacy . 
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PHASE  I 

VOICE  MESSAGE  SUSVET 


The  intent  of  the  questionnaire  survey  was  to  collect 
F/FB/EF-111  crew  members'  preference  data  related  to  specific 
F/FB/EF-111  VMU  design  issues.  In  order  to  generalize,  the 
survey  also  collected  preference  data  on  several  topics 
related  to  generic  voice  warnings  and  cautions. 

Method 


Sub j  ects 

The  CSDF  distributed  150  surveys  to  the  user  population 
(75  TAG  and  75  SAC) ,  of  which  120  were  returned  (a  response 
rate  of  80  percent).  The  data  from  one  of  the  120  subjects, 
who  had  previously  experienced  voice  messages  in  a  cockpit 
environment,  were  deleted  from  the  analysis  to  avoid  biasing 
the  results.  Flying  experience  ranged  from  550  to  4750,  with 
a  mean  of  2104  F/FB/EF-111  flying  hours. 


Apparatus 

The  questionnaire,  titled  *F/FB/EF-111  Digital  Flight 
Control  System  --  Voice  Warning  Survey,"  was  comprised  of 
three  main  data  collection  sections.  A  copy  of  the  survey  is 
shown  in  Appendix  A.  The  first  section  elicited  subjects’ 
personal  information  used  to  establish  the  sample  pool. 

The  second  main  section  of  the  survey  collected  data  relevant 
to  the  four  caution  and  warning  messages  which  were  of 
interest  to  the  proposed  F/FB/EF-lll  VMU:  (1)  GCAS  warning; 

(2)  SCP  violation  warning;  (3)  AOA/SS  caution;  and  (4) 
autopilot  malfunction  caution.  Questions  asked  were  related 
to  the  appropriateness  of  each  message,  and  the  frequency 


of  ropetition  of  that  mosaas*.  The  third  aeotion  of  the 
queationnalre  aaked  the  aubjecta  about  general  voice  related 
topica.  The  apecific  queatlona  elicited  data  related  to 
the  prioritization  of  the  F-111  meaaagea,  preference  of 
voice  gender,  mode  of  preaentation,  and  propoaed  atall 
meaaagea.  The  aubjecta  were  alao  aaked  to  write  any  commenta 
they  perceived  aa  relevant  to  the  topica  being  diacuaaed. 


Procedure 

The  Syatem  Program  Office  (SPO) .  aimul taneoualy  and 
in  equal  numbera .  mailed  150  copiea  of  the  aurvey  to  the 
following  F/FB/EF-111  SAC  and  TAC  Air  Force  communitiea: 
Cannon  AFB;  Plattaburg  AFB;  Langley  AFB;  and  Mountain  Home 
AFB.  Completed  copiea  were  returned  to  the  SPO  within  aix 
weeka  of  the  mailing  date.  Upon  the  return  of  the  copiea,  a 
data  baae  waa  created  to  hold  all  the  information  collected 
from  the  crew  membera  for  aubaequent  data  reduction  and 
atatlatical  analyala. 


Deaian 

The  firat  Phaae  of  the  experiment  waa  dealgned  to 
compare  frequency  data  aa  a  function  of  the  number  of  optiona 

O 

provided  for  aelection.  One-way  Chi  Square  (X**)  analyaea 
were  computed  for  each  of  the  queatlona  aaked.  Reaulta  with 
confidence  probability  levela  of  .95  were  conaidered  to  be 
aigni f leant . 
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Survey  Besulta  end  Disciiaaion 


Individual  one-way  Chi  Square  tests  were  performed  to 
analyze  the  subjective  reports  for  each  of  the  items  covered 
in  the  voice  survey.  For  each  question,  response  frequency 
as  a  function  of  the  available,  or  generated  options  are 
presented  in  a  figure.  Related  F/FB/EF-111  VMU  comments, 
provided  by  the  subjects,  are  listed  in  Appendix  B. 


VMU  Voice  Messages 

GCAS  Warning.  The  frequency  data  for  message  selection  and 
number  of  presentation  are  shown  in  Figures  la  and  lb.  The 
statistically  significant  analysis  for  the  message 

O 

selection  (X  =  111,  p<.05)  implies  that  most  crew  members 
preferred  the  message  PULL-UP  (with  65  out  of  119  votes),  as 
a  warning  for  a  GCAS  warning,  over  the  other  four  messages 
(GROUND  COLLISION;  RECOVER;  ALTITUDE;  and  LOW  ALTITUDE). 

None  of  the  other  messages  were  comparable  to  PULL-UP. 
ALTITUDE  came  in  second  with  only  19  votes. 

An  inspection  of  Figure  lb  suggests  that  most  crew 
members  preferred  that  the  GCAS  warning  be  presented  until 
some  type  of  a  manual  response  is  performed  by  the  crew. 
Statistically  significant  Chi  Square  results  were  found  in 
the  analysis  of  the  frequency  data,  X  =  48,  p<.05. 

SCP  Violation  Warning.  The  Chi  Square  analysis  on  the  SCP 
violation  voice  message  data  (shown  in  Figure  2a)  resulted  in 
a  statistically  significant  difference  between  the  five 
available  options  (ALTITUDE;  A-LOW;  PULL-UP;  TF-LOW;  and  LOW 
ALTITUDE) ,  with  TF-LOW  accumulating  the  highest  frequency 
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la.  Frequency  as  a  function  of  proposed  voice 
messa<{es  for  the  GCAS  warning. 


NUMBER  OF  PRESENTATIONS 


1 b •  Frequency  as  a  function  of  the  number  of 
presentations  for  the  GCAS  warning. 
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FREQUENCY  •  FREQUENCY 


Figure  2b. 


of  votes  (62).  The  Chi  Square  value  (105)  was  Significant  at 
the  .95  confidence  level. 

Similar  to  the  GCAS  warning,  most  crew  members 
preferred  that  this  warning  be  presented  until  the  pilot 
makes  a  response  (i.e.  flies  above  83%  of  the  SCP) .  The  data 
are  shown  in  Figure  2b).  Statistically  significant  frequency 
differences  (X^  =  54,  p<.05)  were  found  between  the  four 
available  options. 

AOA/SS  Caution.  The  AOA/SS  caution  data  are  shown  in  Figures 
3a  and  3b.  As  suggested  by  Figure  3a,  a  statistically 
significant  Chi  Square  (X^  =  34.69,  p<.05)  implied  that  the 

message  ALPHA-SIDE-SLIP  was  preferred  by  most  crew  members 
over  the  other  proposed  messages,  A.O.A.  ,  ALPHA,  and 
A.O.A.S.S.,  with  44  votes.  16  of  the  119  subjects  decided 
not  to  select  any  of  the  available  options,  while  stating  in 
the  comments  section  that  they  believed  the  AOA/SS  caution 
was  not  necessary.  A  complete  list  of  all  the  pilots’ 
comments  is  shown  in  Appendix  B. 

When  asked  about  their  preference  in  the  presentation 
format,  the  crew  members’  questionnaire  data  (shown  in  Figure 
3b)  indicated  that  presenting  the  AOA/SS  caution  message 
three  times  would  be  preferred.  The  statistical 
analysis  resulted  in  significant  differences  between  the 
four  available  options  (X^  =  34.69,  p<.05). 

Autopilot  Failure  Caution.  The  Autopilot  Failure  Caution 
proved  to  be  the  only  question  to  produce  unclear  results 
(data  are  shown  in  Figure  4a).  A  statistically 
significant  Chi  Square  (X^  =  88.5,  p<.05)  indicates  a  clear 
preference  between  the  message  selected  with  the  highest 
frequency,  AUTOPILOT-FAILED,  and  the  two  lowest  frequency 
messages,  AUTO-FAIL  and  AUTO- MALFUNCTION.  However  the  second 
highest  selected  message,  AUTOPILOT,  had  only  7  votes  less 
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Figure  4b.  Frequency  as  a  function  of  the  number  of 


presentations  for  tlie  autopilot  failure  ca 


than  AUTOPILOT-FAILED  (a  difference  of  only  6.6  percent; 
Therefore,  it  will  be  assumed  that  no  realistic  difference  in 
message  preference  exists  between  AUTOPILOT  and  AUTOPILOT 
FAILED. 

Also  unclear,  was  the  frequency  of  presentation 
selection  made  by  the  crew  members  (data  are  shown  in  Figure 
4b).  Again,  while  a  significant  Chi  Square  analysis  was 

Q 

computed  (X  =  10.8,  p<.05),  the  crew  members’  data  were  not 
clear  in  indicating  their  preference  between  two  and  three 
repetitions.  As  with  the  message  selection  data,  it  will  be 
assumed  that  the  crew  members  do  not  have  a  strong  preference 
between  presenting  this  caution  two  or  three  times. 


General  Voice  Related  Items 

Prioritization.  When  asked  to  prioritize  the  warning  and 
caution  messages,  the  crew  members  had  definite  ideas  on  the 
order  of  message  presentation.  An  inspection  of  the  data, 
shown  in  Figure  5,  suggests  the  following:  Most  crew  members 
would  like  for  the  GCAS  warning  to  take  precedence  over  all 
other  voice  message  presentations  (X^  =  256,  p<.05).  The  SCP 
violation  warning  was  ranked  as  second  priority  (X  =  201, 
p<.05),  with  the  AOA/SS  caution  as  third  (X^  =  165,  p<.05), 
and  finally  the  autopilot  failure  caution  as  fourth  = 

233,  p<.05}.  It  also  appears  that  the  crew  members  prefer  to 
prioritize  the  warning  messages  (GCAS  and  SCP  violation) 
higher  than  the  caution  messages  (AOA/SS  and  Autopilot 
failure).  The  warning  messages  were  selected  as  first  or 
second  priority  94  percent  of  the  time  and  as  third  or  fourth 
priority  only  six  percent  of  the  time.  The  caution  selections 
were  reversed:  They  were  expressed  as  first  or  second  only 
six  percent  of  the  time,  while  third  and  fourth  94  percent  of 
the  time. 
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Warning  Mode.  As  shown  in  Figure  6,  it  appears  that  most 
crew  members  indicated  a  preference  for  the  voice  warning 
mode  over  light  and  tone.  The  frequency  analysis  resulted  in 
significant  differences  between  the  three  proposed  options 
(X^  =  36.3,  p<.05). 

It  should  be  noted  in  this  section  that  the  idea  of 
modal  redundancy  was  not  part  of  the  data  being  elicited. 

It  is  well  documented  that  a  warning  presented  by  a  light 
and  accompanied  by  a  voice  message  will  increase  its 
probability  of  being  detected  by  the  user.  For  instance, 
if  a  pilot  is  looking  and  processing  information  outside 
the  cockpit,  he  or  she  may  miss  a  visual  cue,  but  not  an 
auditory  one.  On  the  other  hand,  if  the  auditory  channel 
is  overloaded  with  information,  a  pilot  may  miss  a  voice 
message  but  still  be  able  to  detect  a  visual  cue 
(Werkowitz,  1979). 

Stall  Warning  Mode.  When  given  the  option  as  to  voice  or 
tone  as  a  warning  for  a  stall  conditions,  there  is  no  doubt 
about  which  mode  the  crew  members  prefer.  The  data,  shown  in 
Figure  7,  show  the  tone  warning  as  favored  over  a  voice 
warning.  Aga  in  the  statistical  analysis  verified  the  above 
statement  with  a  significant  difference  between  tone  and 
voice  (X^  =  42.2,  p<.05). 

It  should  be  stated,  however,  that  the  sample  of 
subjects  used  for  data  collection  may  be  bia.oed,  in  that  none 
of  them  had  previous  experience  with  a  stall  voice  warning. 
Despite  their  bias,  the  crew  members  seem  to  like  what  they 
currently  have  on  the  aircraft  and  would  like  to  keep  the 
tone  as  the  warning  mode  for  a  stall  condition. 

Stall  Message  Preference.  In  case  the  stall  warning  would  be 
presented  by  a  voice  message,  the  crew  members  would  prefer 
the  message  STALL  over  RECOVER  (the  data  are  shown  in  Figure 
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0) .  The  Chi  Square  analysis  was  statistically  significant 
(X^  =  49.8,  p<.05). 

Voice  Qender  Preference.  The  preference  frequency  data  on 
the  gender  of  the  voice  message  are  shown  in  Figure  9. 

Almost  86  percent  of  the  subjects  would  prefer  hearing  a 
female  rather  than  a  male  voice  presenting  the  message.  The 

O 

statistical  analysis  was  found  to  be  significant  (X  =  58.9, 
p<.05).  This  may  be  related  to  the  fact  that  most  voices  on 
the  air  waves  are  of  male  speakers,  and  the  female  voice 
provides  the  crew  with  a  distinctive  voice  which  is  different 
from  all  others.  However,  this  distinctiveness  factor  may 
not  be  everlasting,  since  more  women  are  becoming  part  of  the 
Air  Force's  fighting  regime. 
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PHASE  II 

VOICE  STUDY  SIMULATION 


The  intent  of  the  second  phase  of  this  study  was  to 
have  operational  F-111  pilots  experience  the  presentation  of 
the  proposed  voice  messages  in  a  cockpit  environment,  and 
provide  the  SPO  with  feedback,  as  to  their  preference,  on 
some  design  options  relevant  to  the  newly  proposed  VMU. 


Method 


Subi ects 

Six  TAG  and  three  SAC  operational  pilots  participated 
in  the  simulation  study.  An  attempt  was  made  to  insure  that 
the  pilots  were  representative  of  the  overall  population  by 
varying  the  age  and  the  level  of  flying  experience.  The  age 
of  the  pilots  ranged  from  26  to  45  with  a  mean  of  32.5,  and  a 
standard  deviation  of  7.2  years.  Flying  experience  varied 
from  120  to  4100  total  hours,  with  a  mean  of  1970  and  a 
standard  deviation  of  1347  hours.  The  sample  of  pilots 
represented  four  F-111  Air  Force  bases:  Plattsburg  AFB  (SAC, 

2  pilots);  Mountain  Home  AFB  (TAC,  3  pilots);  Cannon  AFB 
(TAG,  3  pilots);  and  Pease  AFB  (SAC,  1  pilot). 


Apparatus 

Facility.  The  study  was  conducted  at  the  Crew  Station  Design 
Facility  (CSDF) ,  which  is  a  U.S.  Air  Force  simulation 
facility  that  belongs  to  the  Aeronautical  Systems  Division 
(ASD)  of  Air  Force  Systems  Command.  The  CSDF  government 
personnel  are  assigned  by  the  Human  Factors  Branch 
(ASD/ENECH)  .  The  facility  (shown  in  Figure  10)  is  used  to 
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conduct  human  anginaarln^  and  aystom  dasign  machanization 
atudlaa  in  support  of  a  variaty  of  System  Program  Offices. 

Simulator.  A  photograph  of  the  FB- 1 1 1  simulator  is  shown  in 
Figure  11.  The  simulator  is  resting  on  a  three  degree  of 
freedom  motion  platform  (not  used  in  current  study),  and 
includes  such  major  components  as  the  control  loading 
assemblies,  seats,  canopies,  and  visual  window.  An  Advanced 
Simulator  Technology  interface  cabinet  is  mounted  on  the  nose 
of  the  cockpit  to  interface  the  simulator  with  the  Gould 
computer  complex.  A  walk-way  with  railings  is  installed 
around  the  simulator  to  insure  pilots’  safety  and  allow  for 
aasy  access  in  and  out  of  the  cockpit.  The  majority  of  the 
instrumentation  on  the  left  aide  of  the  cockpit  are  available 
to  the  pilot.  The  FB-111  software  package  contains  all 
flight,  engine,  atmosphere,  weights,  balances  modules,  and  a 
data  dictionary  of  all  FB-111  data  variables,  as  well  as 
several  other  specific  commons  and  datapools.  All  Linear 
Function  Interpolation  curves  are  validated,  and  all  the 
Avionics  Modernization  Program  (AMP)  configured  pages  for  the 
Central  Display  Unit  (CDU)  and  the  Multi  Function  Displays 
(MFD)  are  incorporated  into  the  software  package. 

Computer  Complex.  The  simulator  is  connected  to  a  series  of 
large  and  small  computer  systems.  This  computer  complex 
includes  five  Gould  series  32/7780,  one  Gould  concept 
32/8780,  two  PDP  11/34,  three  PDP  11/35,  and  two  Silicon 
Graphics  Iris  Computer  Aided  Design  Stations  (one  Iris  2400, 
and  one  Iris  3020). 

Experimenter’s  Console.  The  experimenter’s  console,  shown  in 
Figure  12,  is  located  approximately  12  feet  away  from  the 
simulator.  It  Includes  a  complete  intercom  system,  together 
with  communication  to  and  from  the  pilot  inside  the 
simulator.  The  console’s  displays  duplicate  the  simulator’s 
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visual  Scene,  instruments,  and  displays,  and  are  used  to 
monitor  the  pilot’s  performance.  Furthermore,  the  console’s 
controls  permit  the  experimenter  to  start,  stop,  and  reset 
the  simulation  at  any  time. 

TABLE  1.  The  eight  voice  messages  that  were  evaluated  in  the 
simulator  (alon<t  with  the  time  it  took  to  present 
each  voice  messa<;e)  . 


FAILURE 


GCAS 


SCP  VIOLATION 


AOA/SS 


AUTOPILOT  FAILURE 


MESSAGE 

PULL-UP 

CLIMB 

TF-LOW 

ALTITUDE 

ALPHA  SIDE  SLIP 

ALPHA 

AUTOPILOT 

AUTOPILOT  FAILED 


TIME  (in  seconds) 

0 . 420 

0 . 506 

0 . 850 

0 . 830 

1 . 050 

0.510 

0 .720 

1 . 000 


Voice  Messages.  Eight  voice  messages,  shown  in  Table  1,  were 
recorded  on  an  Amiga  micro  computer  by  a  female  employee  of 
the  CSDF.  The  employee,  who  had  a  distinctive  and  mature 
mid-western  voice,  presented  the  messages  in  a  formal  and 
impersonal  manner.  The  Amiga  used  a  high  speed  voice 
digitizer  (Future  Sounds),  with  a  sampling  rate  of  10,000 
samples  per  second,  to  convert  the  messages  from  analog 
to  digital  format.  The  Amiga  was  thereafter  connected  to 
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the  main  frame  computers  using  an  RS-232  interface,  and 
transmitted  the  messages  to  the  pilot's  head  set  (an 
ASTROCOM  model  number  20680  with  MX-2508/A/C  pads) 
through  the  intercom  channel.  The  length  of  time  it  took 
to  articulate  each  message  ranged  from  0.42  to  1.05 
seconds.  The  time  data  are  also  shown  in  Table  1. 

Audio  Systems.  Background  communication  was  simulated  by  an 
audio  tape  which  was  replayed  throughout  half  the  missions 
of  the  experiment,  on  a  Technics  by  Panasonic  model  number 
RS-26ZAUS  tape  player,  and  transmitted  to  the  pilot’s  head 
set  through  the  intercom  channel. 


Procedure 


The  pilots  were  instructed  to  fly  the  simulator,  head 
down,  either  in  a  manual  or  an  automatic  Terrain  Following 
(TF)  mode.  During  manual  TF  missions,  the  pilots  encountered 
GCAS  and  SCP  violation  warnings,  while  the  AOA/SS  and 
Autopilot  Failure  cautions  were  heard  throughout  the  auto  TF 
missions.  Each  time  a  voice  message  was  activated,  the 
pilots  were  instructed  to  immediately  deactivate  the  failure 
by  manually  pressing  the  appropriate  switch  located  on  the 
flying  stick.  The  SCP  and  Autopilot  failure  were  responded 
to  by  pressing  the  paddle  switch,  while  GCAS  and  AOA/SS 
failures  were  deactivated  by  pressing  the  trigger. 

Throughout  half  the  missions,  background  communication 
was  presented  by  playing  an  audio  tape  of  a  combat  mission, 
recorded  during  the  Vietnam  war,  and  transmitted  to  the  pilot 
through  the  head  set.  The  pilots  were  Instructed  to  ignore 
background  communication,  which  was  irrelevant  to  the  success 
of  their  mission. 

Each  pilot  flew  one  practice  run  of  each  of  the  four 
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experimental  missions,  and  was  exposed  to  all  eight  voice 
messages.  The  pilot  then  flew  the  same  four  missions  with 
data  collection  turned  on.  Each  mission  lasted  approximately 
12  minutes.  The  warning  and  caution  messages  were  presented 
to  the  pilot  at  an  average  of  30  seconds,  with  a  standard 
deviation  of  10  seconds,  for  a  total  of  24  events  (six 
repetitions  per  message).  At  the  completion  of  the  flying 
missions,  the  pilots  were  asked  to  complete  a  short  survey 
relevant  to  the  voice  messages  being  evaluated  and  to  the 
fidelity  of  the  simulator.  Each  pilot  participated  in  the 
study  for  approximately  three  hours,  which  included  a  15 
minute  break  between  replications. 


VMU  Mechanization 


All  eight  messages  were  presented  to  the  pilot  through 
the  intercom  channel  at  approximately  90  dB.  In  order  to 
maintain  consistency  across  subjects  and  conditions,  pilots 
were  told  not  to  change  the  volume  control  setting.  Warning 
messages  (Pullup,  Climb,  Altitude,  and  TF-Low)  were  mechanized 
in  such  a  way  that  the  messages  were  continuously  presented 
to  the  pilot  until  the  corrective  action  was  completed,  with 
an  inter-message  interval  of  500  milliseconds.  The  caution 
messages,  on  the  other  hand,  were  also  presented  with  an 
inter-message  interval  of  500  milliseconds,  except  that  the 
messages  were  always  repeated  three  times  irrespective  of 
the  status  of  the  response.  This  format  was  influenced  by 
the  results  found  in  the  first  phase  of  this  evaluation. 

The  format  makes  sense  in  that  normally,  warnings  require 
some  type  of  an  immediate  manual  response,  whether  the  pilot 
is  actively  pressing  a  button  or  pulling  on  the  flying  stick. 
On  the  other  hand,  cautions,  which  may  still  be  critical  to 
the  operation  of  the  aircraft,  do  not  always  require  a 
corrective  action. 
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Th«  •xperiment  waa  daalgned  to  compara  pilota’ 
objective  and  aubjective  data  to  either  of  two  neaaagea  per 
warning.  The  objective  data  included  reaponae  time,  from 
onaet  of  the  meaaage  until  the  trigger  or  paddle  awitch  waa 
preaaed,  and  reaponae  accuracy.  Reaponae  time  calculationa 
were  independent  from  the  meaaage  preaentation  time.  Alao  of 
intereat  to  thia  evaluation  waa  the  effect  of  background 
communication  on  the  comprehenaion  and  recognition  of  the 
meaaagea.  Therefore,  throughout  half  the  miaaiona, 
background  communication  waa  preaented  aa  an  auditory 
interference  and  general  noiae  factor. 

The  aubjective  data  (gathered  through  the 
queationnaire)  elicited  preference  aelectiona  between  the  two 
meaaagea  (per  failure)  that  were  experienced  throughout  the 
atudy.  The  pilota  were  alao  aaked  to  aubjectively  evaluate 
the  location  and  nomenclature  of  the  new  warning  and  caution 
lighta  that  correaponded  with  the  voice  meaaagea. 


Siaiulator  Reaulta 


Objective  Data 

Pilota'  mean  correct  reaponae  time  and  percent  reaponae 
accuracy  data  were  analyzed  uaing  four  aeparate  2X2  two  way 
repeated  meaaurea  analyaea  of  variance  (ANOVA) ;  one  ANOVA  per 
failure  condition.  The  Independent  varlablea  were  voice 
meaaage  (two  meaaagea  per  failure  condition)  and  background 
communication  (ON  veraua  OFF).  The  reaponae  accuracy 
analyaea  were  performed  in  order  to  teat  for  poaaible 
reaponae/accuracy  trade-offa.  Throughout  the  reaulta 
aection,  only  atatiatical ly  aignificant  (p  <  .05)  F  valuea 
will  be  reported. 
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TABLE  2.  Pilots’  mean  response  time  and  percent  response 
accuracy  for  the  GCAS  failure. 

RESPONSE  TIME 


BACKGROUND 

COMMUNICATION 

ON 

OFF 

MEAN 

CLIMB 

1 . 086 

0.959 

1 . 023 

PULLUP 

1 . 038 

1 .098 

1 .068 

MEAN 

1 . 062 

1.029 

PERCENT  ACCURACY 

BACKGROUND 

COMMUNICATION 

ON 

OFF 

MEAN 

CLIMB 

94 

94 

94 

PULLUP 

98 

93 

95 

MEAN 

96 

94 
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GCAS  Warning.  An  inspection  of  the  GCAS  means,  shown  in 
Table  2,  indicates  that  pilots’  response  times  were  affected 
neither  by  the  voice  message  nor  by  the  background 
communication.  The  analysis  of  variance  resulted  in  a  non¬ 
significant  two-way  interaction  as  well  as  non-significant 
main  effects  of  message  and  background  communication. 

The  response  accuracy  data  coincided  with  the  response 
time  data,  in  that  no  significant  interaction  or  main  effects 
resulted  from  the  analysis. 

SCP  Warning.  The  means  for  the  SCP  violation  warning  are 
shown  in  Table  3.  The  response  time  ANOVA  resulted  in  a 
statistically  significant  main  effect  of  background 
communication  (F(l,a)  =  6.66,  £  <  .05).  An  inspection  of  the 
means  in  Table  3  indicates  that  pilot  response  times  were 
119  ms  slower  when  background  communication  was  ON  in 
comparison  with  no  background  communication  (OFF).  Non¬ 
significant  differences  were  identified  for  the  two-way 
interaction  as  well  as  the  main  effect  of  voice  message. 

Despite  a  10  percent  accuracy  advantage  for  TF-LOW  over 
ALTITUDE,  the  means  were  not  found  to  be  significant. 

However,  the  standard  deviations  for  the  two  messages  were 
calculated  at  39  and  29  percent.  These  differences  were  large 
enough  to  negate  any  indication  for  a  possible  trend.  The 
response  accuracy  data  offered  no  significant  differences  for 
the  interaction  or  the  main  effect  of  background  communication. 

AOA/SS  Caution.  The  analysis  of  variance  on  pilot  response 
time  resulted  in  a  significant  main  effect  of  voice  message, 
F(l,8)  =  7.42,  £  <  .05.  An  inspection  of  the  means,  shown  in 
Table  4,  indicates  that  the  pilots  responded  faster  to  ALPHA 
(1.096  seconds)  than  to  ALPHA  SIDE  SLIP  (1.189  seconds).  The 
two  way  interaction  and  the  main  effect  of  background 
communication  were  not  found  to  be  statistically  significant. 
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TABLE  3. 


Pilots*  mean  response  time  and  percent  response 
accuracy  for  the  SCP  failure. 

RESPONSE  TIME 

BACKGROUND  COMMUNICATIQM 


ON 

OFF 

MEAN 

ALTITUDE 

1  .  176 

1 . 076 

1.126 

TFLOW 

1  .  222 

1 . 087 

1.154 

MEAN 

1 . 200 

1.081 

PERCENT  ACCURACY 

BACKGROUND  COMMUNICATION 


ON 

OFF 

MEAN 

ALTITUDE 

83 

79 

81 

TFLOW 

93 

89 

91 

MEAN 

88 

84 
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TABLE  4.  Pilots*  mean  response  time  and  percent  response 
accuracy  for  the  AOA/SS  failure. 


RESPONSE  TIME 

BACKGROUND  CQMMUNICATIOM 


ON 

OFF 

MEAN 

ALPHA 

1  .  138 

1 .054 

1 . 096 

ALPHA 

SIDE 

SLIP 

1.157 

1 . 222 

1  .  189 

MEAN 

1  .  147 

1  .  138 

PERCENT 

ACCURACY 

BACKGROUND 

COMMUNICATION 

ON 

OFF 

MEAN 

ALPHA 

93 

96 

94 

ALPHA 

SIDE 

SLIP 

91 

98 

94 

MEAN 

92 

97 
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The  response  accuracy  ANOVA  identified  no  statistically 
significant  interaction  or  main  effects. 

Autopilot  Failure  Caution.  The  complete  list  of  means  is 
shown  in  Table  5.  The  results  of  the  ANOVA  for  pilot 
response  time  data  identified  a  significant  two-way 
interaction  between  voice  message  and  background 
communication  (F(1,0)  =  9.26,  p.  <  .05).  An  inspection  of 
Figure  13  suggests  that  while  no  difference  in  response  time 
data  seems  to  exist  between  the  two  messages,  when  background 
communication  was  off,  the  background  communication  "ON" 
condition  affected  pilot  performance  in  such  a  manner  that 
response  time  data  were  found  to  be  faster  to  the  message 
AUTOPILOT  (1.048  seconds)  than  to  the  message  AUTOPILOT 
FAILED  (1.163  seconds).  The  results  of  the  response  time 
ANOVA  were  non-significant  for  the  main  effects  of  voice 
message  and  background  communication. 

As  with  the  previous  accuracy  data,  neither  the  two-way 
interaction  nor  the  main  effects  were  found  to  be 
statistically  significant  throughout  the  analysis  of 
variance . 
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Figure  13.  Pilots’  mean  response  time  data  as  a  function  of 
voice  message  and  background  communication 
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BACKGROUND  COMMUNICATION 
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TABLE  5. 


Pilots’  mean  response  time  and  percent  response 
accuracy  for  the  autopilot  failure. 


RESPONSE  TIME 


background 

COMMUNICATION 

ON 

OFF 

MEAN 

AUTOPILOT 

1 . 048 

1 . 069 

1 . 059 

AUTOPILOT  FAILED 

1  .  163 

1 . 048 

1  .  105 

MEAN 


I 

j 

1.106  1.059 


PERCENT  ACCURACY 

BACKGROUND  COMMUNICATION 


ON 

OFF 

MEAN 

Jt 

AUTOPILOT 

89 

87 

88 

AUTOPILOT  FAILED 

96 

85 

91 

MEAN 

96 

86 
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Subjective  Data 


Due  to  the  small  number  of  pilots  used  throughout  the 
simulation  effort,  it  was  decided  by  the  experimenter  that 
a  formal  statistical  analysis  using  Chi  Square  would  not  be 
appropriate.  However,  for  the  sake  of  data  interpretation,  a 
frequency  totaling  two  thirds  of  the  total  sample  (six 
pilots)  would  be  considered  a  majority  opinion. 

The  frequency  data  for  the  preferred  voice  messages 
are  shown  in  Figures  14  through  17.  The  bar  graphs 
indicate  a  majority  preference  for  the  following  voice 
messages:  TF-LOW  for  SCP  violation  warning,  and  ALPHA 

SIDE  SLIP  for  AOA/SS.  No  clear  preference  was  indicated 
for  the  GCAS  warning  or  the  Autopilot  Failure  caution. 

The  following  paragraph  reflects  other  subjective  data 
relative  to  the  mechanization  of  the  F/FB/EF-111  VMU  (The 
data  are  shown  in  Figures  10  through  20):  Six  of  the  pilots 
confirmed  the  need  for  a  redundant  GCAS  mode  (such  as  light); 
eight  out  of  nine  pilots  believed  that  the  500  milliseconds 
inter-message- ti me  was  optimum  (pilot  nine  expressed  concern 
that  the  inter-message- interval  was  too  long);  the  same  eight 
out  of  nine  pilots  believed  that  the  three  time  caution 
repetition  was  optimum  (pilot  nine  recommended  to  decrease 
the  number  of  presentations);  all  nine  pilots  responded 
affirmatively  when  asked  about  the  nomenclature  and  location 
of  the  AOA/SS  and  the  FCS  caution  lights  indicating  an 
Autopilot  Failure.  Finally,  all  nine  pilots  believed  that 
the  use  of  the  Radar  Alt  Low  light  (flashing)  as  a  warning 
indicator  for  an  SCP  violation  is  acceptable.  > 

Comments  elicited  from  the  pilots  in  regard  to  the 
above  data  are  listed  in  Appendix  B. 
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Figure  17. 


Figure  18. 


Figure  19. 
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IS  REPETITION  OPTIMUM  ? 


Figure  20.  Pilots*  preference  Frequency  as  a  function  of  the 


repetition  of  the  caution  voice  messages. 


GENERAL  DISCUSSION  AND  CONCLUSION 


The  voice  message  system  in  the  F-111  aircraft  should 
provide  a  crucial  capability  to  the  crew  in  the  areas  of 
workload  reduction  and  safety  in  flight.  It  is  intended  that 
the  production  of  the  system  provide  highly  acceptable 
results  realizing  the  inevitable  tradeoffs  between  system 
quality  and  computer  memory.  The  results  of  this  evaluation 
as  well  as  the  system  requirements  should  serve  as  a  guide  to 
the  development  of  this  voice  message  unit. 

Werkowitz  (1980)  describes  four  elements  which  are 
necessary  for  an  operator  to  receive  and  use  a  spoken 
message:  1)  Awareness  that  a  message  is  being  presented,  2) 
Recognition  of  the  words  of  the  message,  3)  Understanding  the 
message,  and  4)  Knowledge  of  how  to  use  the  information.  As 
a  result  of  research  and  experience,  many  guidelines  and 
requirements  exist  which  apply  to  the  development  of  a  voice 
message  system  and  can  be  correlated  with  the  above  listed 
elements  and  the  results  of  this  evaluation. 


Message  Awareness 

The  awareness  that  a  message  is  being  presented  relates 
to  the  method  of  presentation  of  the  message.  MIL-STD-1472 
dictates  that  verbal  warning  signals  shall  consist  of  an 
initial  alerting  signal  (non-speech)  to  attract  attention  and 
to  designate  the  general  problem  area.  Due  to  the  response 
time  critical  issue  associated  with  some  of  the  warnings, 
this  requirement  becomes  questionable  and,  in  fact,  is  not 
recommended  for  most  uses  of  verbal  warnings.  The  standard 
also  states  that  the  voice  used  in  recording  verbal  warning 
signals  shall  be  distinctive  and  mature.  Typically  in  voice 
message  systems,  a  female  voice  has  been  used  for  two  main 


reasons;  (1)  Due  to  its  higher  frequency  than  the  male  voide; 
and  (2)  to  make  it  stand  out  from  radio  communications  which 
are  predominantly  male  voices,  although,  the  influx  of  women 
into  the  services  may  eventually  decrease  the  efficacy  of 
this  approach.  The  evaluation  showed  that  a  female  voice  is 
preferred  by  the  F/FB/EF-111  pilots  surveyed.  MIL-STD-4 1  ID . 
Aircrew  Station  Signals,  dictates  that  verbal  warning  signals 
shall  only  be  used  to  complement  red  warning  or  other 
critical  visual  signals.  This  redundancy  is  evident  in  the 
design  of  this  voice  warning  system.  Each  of  the  cautions 
(AOA/SS  and  autopilot  failure)  are  redundant  to  caution 
lights  on  the  master  caution  panels.  The  SCP  violation 
warning,  in  turn,  is  redundant  to  the  Radar  Altitude  Low 
warning  light.  The  GCAS  warning  represents  a  new  addition  to 
the  aircraft  with  no  visual  redundancy  at  this  time.  This  is 
in  contradiction  with  the  military  specifications  (MIL-STD- 
411),  of  which  the  CSDF  agrees  with.  A  light  signal  should 
be  the  primary  mode  of  presentation,  which  may  then  be 
complemented  by  a  voice  message.  Repetition  of  the 
messages  by  the  warning  system  represents  a  further 
aid  to  awareness.  The  pilots  surveyed  favored  three 
statements  of  the  AOA/SS  caution,  two  or  three 
statements  of  the  autopilot  failure  caution,  and,  due 
to  the  criticality  of  the  warnings,  repetition  of  the 
GCAS  and  SCP  violation  warnings  until  action  is  taken 
by  the  pilot. 


Recognition  of  words 

The  recognition  of  the  words  of  the  message  determines 
the  effectiveness  of  the  communication  system.  This  element 
is  dependent  upon  the  nature  of  the  speech  material;  the 
relevant  characteristics  of  the  talker  and  listener;  the 
conditions  under  which  the  radiation,  transmission,  and 
reception  of  the  speech  wave  occur;  and  the  characteristics 
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of  the  communication  system.  Voice  communication  may  be 
degraded  by  a  variety  of  system  and  environmental  factors 
that  include  electrical  or  acoustical  noise  or  both,  radio 
interference,  jamming,  communication  signal  processing  and 
various  other  factors  that  prohibit  effective  communication. 
Important  in  the  recognition  of  the  words  is  the  intensity 
and  the  intelligibility  of  the  system.  MIL-STD-1472  presents 
requirements  for  intensity,  stating  that  verbal  alarms  for 
critical  functions  shall  be  at  least  20  dB  above  the  speech 
interference  level  at  the  operating  position  of  the  intended 
receiver.  The  speech  interference  level  (SIL)  is  the 
arithmetic  average  of  the  sound  pressure  levels  of  the 
interfering  noise,  in  decibels,  in  the  four  octave  bands 
centered  on  the  frequencies  500,  1000,  2000  and  4000  Hz, 

respectively.  MIL-STD- 1 000 ,  Human  Engineering  Performance 
Requirements  for  Systems,  specifies  three  recommended  methods 
for  assessing  the  speech  intelligibility  of  a  given  system: 

1)  The  ANSI  standard  method  of  measurement  of  phonetically 
balanced  (PB)  monosyllabic  word  intelligibility,  2)  The 
modified  rhyme  test  (MRT)  ,  and  3)  the  articulation  index  (AI) 
calculations . 

Analytical  studies  of  voice  warning  system  performance, 
environmental  influences  and  the  man-in-the-loop  element  must 
be  carried  out  under  carefully  controlled  conditions  that 
simulate  to  the  greatest  extent  possible,  the  practical, 
operational  situations  of  concern.  As  speech  intelligibility 
is  the  most  critical  aspect  of  a  voice  message  system,  it  is 
more  advantageous  to  conduct  laboratory  intelligibility 
testing  with  the  communication  system  in  as  near  an 
operational  configuration  as  possible.  All  listeners  should 
wear  standard  custom  fitted  flight  helmets,  earphones,  and 
oxygen  masks  through  which  they  receive  breathing  air. 
Furthermore,  the  noise  environment  in  which  the  system  is  to 
be  eventually  used  should  be  emulated  as  closely  as  possible. 
In  all  cases  involving  evaluation  of  the  intelligibility 
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and/or  quality  of  a  speech  synthesis  system,  the  human 
listener  is  the  ultimate  discriminator. 


Understanding  of  message 

With  respect  to  the  understanding  of  the  message,  MIL- 
STD-1472  states  that  verbal  warning  signals  shall  consist  of 
a  brief  standardized  speech  signal  (verbal  message)  which 
identifies  the  specific  condition  and  suggests  appropriate 
action.  Also,  in  selecting  words  to  be  used  in  audio  warning 
signals,  priority  shall  be  given  to  intelligibility,  aptness, 
and  conciseness  in  that  order.  It  was  with  these  ideas  in 
mind  that  this  evaluation  was  done,  emphasizing  the  aptness 
and  conciseness  of  the  messages.  From  the  alternatives 
offered  in  the  questionnaire,  for  the  various  messages, 
preferred  messages  were  determined.  PULL-UP  was  chosen  for 
the  QCAS  warning  with  ALTITUDE  as  a  not  so  close  second.  In 
the  simulation  study,  PULL-UP  was  evaluated  against  CLIMB,  a 
write-in  choice  by  a  few  of  the  respondents,  with  no 
significant  results.  This  was  done  to  allow  for  ALTITUDE  to 
be  evaluated  as  the  SCP  violation  warning.  Headquarters 
Tactical  Air  Command  has  issued  the  recommendation  that  PULL 
UP  be  used  for  the  GCAS  warning,  and  it  is  intended  to  be, 
with  specific  exceptions.  For  the  take-off  and  landing  modes 
of  the  aircraft,  pulling  up  may  be  the  incorrect  maneuver  and 
result  in  a  stall  or  crash.  For  this  application,  a 
different  message,  i.e.  ALTITUDE  or  CLIMB,  might  be  more 
appropriate.  TF-LOW  was  chosen  for  the  SCP  violation  warning 
with  PULL-UP  and  ALTITUDE  as  a  not  so  close  second  and  third, 
respectively.  In  the  simulation,  TF-LOW  was  evaluated 
against  ALTITUDE,  allowing  for  PULL-UP  to  be  evaluated  as  the 
GCAS  warning,  with  TF-LOW  showing  a  clear  preference  once 
again.  ALPHA  SIDE-SLIP  was  chosen  for  the  AOA/SS  caution  with 
ALPHA  as  second  choice.  The  simulator  evaluation  showed 
ALPHA-SIDE-SLIP  preferred  two  to  one  over  ALPHA.  AUTOPILOT 
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FAILED  was  chosen  as  the  autopilot  failure  caution  with 
AUTOPILOT  as  a  close  second.  The  simulator  evaluation  did 
not  show  a  significant  preference. 


Prioritizing  the  Messages 

It  is  possible  that  two  or  more  of  the  failure 
conditions  may  occur  simultaneously.  In  such  a  condition, 
the  messages  need  to  be  prioritized  so  that  the  most  critical 
message  is  presented  first.  If  such  prioritization  is  deemed 
appropriate  for  the  subject  aircraft,  the  following  order 
would  be  most  appropriate:  (1)  GCAS,  (2)  SCP  violation,  (3) 

AOA/SS,  and  (4)  Autopilot  Failure. 


Knowledge  of  Use 

The  knowledge  of  how  to  use  the  information  in  the  case 
of  pilots  relates  to  training  in  the  aircraft.  MIL-STD-4 1  ID , 
Aircrew  Station  Signals,  dictates  that  verbal  warning  signals 
shall  be  audible  signals  in  verbal  form  indicating  the 
existence  of  a  hazardous  or  imminent  catastrophic  condition 
requiring  immediate  action.  The  pilots  should  be  thoroughly 
trained  in  the  response  methodology  to  such  warnings. 
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APPENDIX  A 


VOICE  WARHINQS  SURVEY 


The  following  survey  deals  specifically  with  the  voice 
warning  messages  that  will  be  implemented  in  the  cockpit  of 
the  F/FB/EF-111  aircraft  as  part  of  the  new  Digital  Flight 
Control  System  (DFCS) .  Your  serious  cooperation  in  the 
completion  of  the  survey  will  insure  that  the  incorporated 
voice  messages  will  be  most  consistent  with  the  flight  crew 
population's  terminologies  and  definitions  of  the  relevant 
warnings.  Furthermore,  the  results  of  the  survey  will  be 
used  in  man- in-the-loop  simulation  at  the  Crew  Station  Design 
Facility  (ASD/ENECH-CSDF,  Wright  Patterson  AFB)  in  order  to 
experimentally  evaluate  the  voice  messages.  This  in  turn 
could  insure  optimal  crew  members’  performance,  with  faster 
response  times  and  lower  levels  of  workload  demand. 

For  each  of  the  following  multiple  choice  questions, 
circle  what  you  think  would  be  the  most  appropriate  option. 
For  each  question,  an  additional  section  is  provided  so  you 
may  include  comments  which  are  relevant  to  the  topic  in 
question.  This  section  will  also  allow  you  to  add  any 
additional  options  (such  as  another  voice  message)  which  were 
not  previously  listed.  However,  make  sure  you  select  one 
of  the  available  options  before  you  include  any  comments. 

Please  use  your  personal  Judgment  in  responding  to  all 
the  items  covered  throughout  the  survey.  All  surveys  should 

be  completed  and  returned  to  _ ,  at  _ ,  by 

16  November,  1987. 
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PERSONAL  DATA: 


NAME: _ 

RANK : _ 

AERO  RATING: _ 

ORGANIZATION : _ 

OFFICE  SYMBOL: _ 

DUTY  STATION: _ 

FLYING  HOURS: _ 

DESCRIBE  ANY  PRIOR  EXPERIENCE  (IF  ANY)  WITH 
VOICE  WARNING  MESSAGES: _ 
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Voice  warning  questions 


A.  GROUND  COLLISION  AVOIDANCE  SYSTEM  (GCAS).  This  voice 
warning  will  be  capable  of  providing  the  necessary 
enunciation  to  effectively  alert  the  crew  of  an 
impending  ground  collision. 

1.  Which  of  the  following  voice  messages  would  you  consider 
to  be  most  appropriate  for  the  GCAS  warning  ? 

(a)  Pull-Up 

(b)  Ground-Collision 

(c)  Recover 

(d)  Altitude 

(e)  Low  Altitude 


COMMENTS: 


2.  How  many  times  should  the  GCAS  warning  message  be  repeated 
with  each  incident 

(a)  Once 

(b)  Twice 

(c)  Three  times 

(d)  Until  the  conditions  to  affect  the  recovery 
are  completed 

COMMENTS ; _ 
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B.  AOA/SS  FAILURE  WARNING.  (Angle  Of  Attack/SideSl ip)  This 
caution  will  indicate  that  there  is  a  performance  degradation 
in  AOA  limiter  and/or  adverse  yaw  compensation.  The 
specifics  of  the  cause  are  attributable  to  one  of  the 
following  failures: 

Airborne:  *  AOA  Probe  Miscompare 

*  Dual  Probe  Transmitter  Failure 

»  Dual  Probe  Heater  Failure 

Dual  Roll  Rate  Failure 
»  Senes  Trim  Actuator  Failure 
»  Total  Aileron  Position  Failure  Flaps  Up 

*  Large  Initial  SS  or  Flight  Control  Disconnect 
with  Flaps  Down 

Ground:  »  Probe  Heat  is  Deselected 

»  Probe  is  Selected  and  Heaters  in  either  LAOA, 
RAOA,  or  Beta  have  Overheated 

1.  Which  of  the  following  voice  messages  would  you  consider 
to  be  most  appropriate  for  the  AOA/SS  caution 

(a)  AOA 

(b)  Alpha 

(c)  AOA-SS 

(d)  Alpha-Side-Slip 

COMMENTS  :  _ 
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2.  How  many  times  should  the  AOA/SS  caution  message  be 
repeated  with  each  incident  ? 


(a)  Once 

(b)  Twice 

(c)  Three  times 

(d)  Until  the  pilot  completes  the  corrective  action 

COMMENTS : _ 


c. 

SET  CLEARANCE 

PLANE 

i  s 

ini tiated 

when 

the  a 

selected  SCP 

al  t  i 

tude . 

(SCP)  VIOLATION  (83%). 
ircraft  flies  below  83% 


The  warning 
o  f  the 


1. 


Which 
to  be 


of  th 

e  following  voice  messages  would 

you  consider 

most 

appropriate  for  the  SCP  violation 

warning 

(a) 

Al t i tude 

(b) 

A  Low 

( c ) 

Pull-Up 

(d) 

TF  1  ow 

(e) 

Low  Altitude 

COMMENTS : 


$■ 
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2. 


How  many  times  should  the  SCP  violation  warning  message 
be  repeated  with  each  incident  '> 


(a)  Once 

(b)  Twice 

(c)  Three  times 

(d)  Until  the  aircraft  is  above  837.  of  SCP 


COMMENTS : 


D.  AUTOPILOT  FAILURE  CAUTION.  This  caution  indicates  that 
the  autopilot  is  malfunctioning.  The  failure  may  occur 
either  when  the  pilot  is  engaging  autopilot,  or  while  the 
aircraft  is  already  in  autopilot  mode.  The  failure  will 
cause  the  autopilot  to  automatically  disengage. 


* 


« 


1.  Which  of  the  following  voice  messages  would  you  consider 
to  be  most  appropriate  for  the  autopilot  failure  caution 

(a)  Autopilot  Failed 

(b)  Autopilot 

(c)  Auto  Fail 

(d)  Auto  Malfunction 

COMMENTS  : _  _ 
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2. 


How  many  times  should  the  autopilot  failure  caution 
message  be  repeated  with  each  incident  '> 


(a)  Once 

(b)  Twice 

(c)  Three  times 

COMMENTS  : 


> 


E.  GENERAL  VOICE  WARNINGS.  The  following  questions  will 
discuss  the  voice  message  mechanization  process  in  general. 


1.  Which  of  the  failures  should  take  precedence,  in  case 

two  or  more  of  them  need  to  be  simultaneously  presented 
to  the  pilot.  Rank  the  failures  in  the  prioritized 
order  that  you  consider  to  be  optimum. 

(  )  GROUND  COLLISION  AVOIDANCE  SYSTEM 
(  )  AOA/SS  FAILURE 
(  )  SCP  VIOLATION 
(  )  AUTOPILOT  FAILURE 

COMMENTS:  _ 
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As  a  general  rule,  which  of  the  following  warning  "mode" 
is  most  effective  in  attracting  crew  members’  attention 
in  a  cockpit  environment  ’ 


(a)  Light 

(b)  Tone 

(c)  Voice 

COMMENTS : 


As  a  general  rule,  Which  voice  format  would  you  consider 
to  be  most  efficient  in  attracting  your  attention  ? 


(a) 

Human 

female  voice 

(b) 

Human 

male  voice 

COMMENTS : 

Which  of  the  following  two  codes  would  you  prefer  to 
represent  a  stall  warning  (the  present  mode  is  tone)  ’ 

(a)  Tone 

(b)  Voice 


COMMENTS : 


5.  If  a  stall  voice  warning  were  installed  in  the  aircraft, 
which  of  the  following  messages  would  you  consider  to  be  the 
most  appropriate? 

(a)  Stall 

(b)  Recover 

COMMENTS : _ 
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APPENDIX  B 


SUBJECTIVE  COMMENTS 

,  PHASE  I  --  SURVEY 

«  A)  The  following  comments  are  relevant  to  the  GCAS  warriing: 

'i  .  Crew  may  know  they  are  temporarily  going  below  GCAS 
warning.  A  non  directive  message  warning  the  crew 
without  giving  a  directive  that  the  crew  does  not  need. 

2.  Climb  is  another  suggestion  for  a  GCAS  warning. 

3.  If  the  pilot  is  disoriented,  PULL-UP  is  a  good 
message.  If  the  pilot  is  not  disoriented,  then  he  should 
know  why  he  is  getting  the  warning. 

4.  The  volume  of  the  GCAS  warning  should  be  directly 
proportional  to  the  severity  of  the  pull  up  required. 

5.  The  appropriate  message  should  be  dependent  on  how 
far  you  are  from  impact. 

6.  PULL-UP,  in  the  heat  of  the  battle,  could  be 
interpreted  as  Pull  Back. 

^  7.  "ROCKS'  is  another  message  for  a  GCAS  warning. 

B)  The  following  comments  are  relevant  to  the  SCP  violation 
warning . 

1.  The  voice  warning  is  not  necessary. 
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2.  Should  be  fairly  high  pitched  voice. 


3.  Low  Altitude/Pul 1-Up . 

C)  The  following  comments  are  relevant  to  the  AOA/SS 
caution . 

1.  Alpha-SIS  is  easier  to  recognize. 

2.  Use  two  different  messages.  'AOA*  for  alpha,  and 
’Side  Slip*  for  SIS. 

3.  Looks  like  too  many  things  setting  off  the  same 
caution.  It’s  giving  the  pilot  little  useful 
information . 

4.  The  caution  is  considered  unnecessary. 

5  We  have  sufficient  lights  for  heater  problems. 

6.  SIS  would  be  best. 

7.  Malfunction  light  is  good  enough. 

8.  Tone  is  better  than  voice  actuator. 

D)  The  following  comments  are  relevant  to  the  Auto  Pilot 
failure  caution. 

1.  Not  necessary  except  for  ATF,  and  the  message  should 
be  AUTO  TF. 

2.  The  only  good  caution  would  be  an  'RNE‘  (Reference  Not 
Engaged)  light. 
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3.  We  really  don’t  need  a  voice  message  for  autopilot. 

4.  Another  message  would  be:  “Modes 
des e 1 ec ted--Autopi lot  failed." 

*  E)  The  following  comments  are  relevant  to  the  prioritization 

issue . 

1.  GCAS  and  SCP  violation  are  equal  priorities, 
depending  on  whether  the  aircraft  is  in  ATF  or  manual 
mode  . 

2.  The  cautions  are  not  needed. 


F)  The  following  comments  are  relevant  to  the  warning  mode 
preference . 

1.  In  the  F-111  A/E,  light  is  more  effective  than  tone. 

2.  Flashing  red  in  a  centrally  located  position  is  best 
indicator . 

3.  A  combination  is  better  since  it  affects  more 
physical  senses. 

4.  Depends  on  conditions. 

5.  Hard  to  judge,  no  previous  experienced  with  voice. 

6.  Tone-light  or  voice-light  combinations. 

7.  Depends  on  situation:  In  a  high  noise  environment, 

light  would  provide  more  information.  However,  in  a  high 
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illumination  condition,  one  more  light  may  not  be 
signi f leant . 

8.  Already  too  many  lights  in  the  FB-111. 

9.  We  already  have  too  many  tones,  lights  depends  on 
location . 

G)  The  following  comments  are  relevant  to  the  gender  mode 
pre  f  er ence . 

1.  A  female  voice  is  more  out  of  the  ordinary  in  the 
fighter  environment. 

2.  A  female  voice  is  distinguishable  from  other  crew 
members . 

3.  Neither  male  nor  female  is  best. 

H)  The  following  comments  are  relevant  to  the  Stall  warning 
mode  preference, 

1.  Already  used  to  tone. 

2.  Tone  only.  Pilots  use  the  tone  to  check  their  turn 
performance.  Do  not  distract  with  a  voice, 

3.  The  plane  isn’t  stalling  at  Stall  warning,  so  voice 
is  irrelevant. 

4.  Don’t  fix  it.  It  ain't  broke. 

5.  The  cockpit  is  already  very  noisy.  Adding 
another  voice  may  be  counterproductive. 
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6.  Suggests  a  two  second  tone,  followed  by  a  voice 
message . 

7.  Do  not  remove  present  tone.  Add  voice  in  addition. 
0.  Too  many  tones.  I  like  voice. 


I)  The  following  comments  are  relevant  to  the  Stall  message 
pre  f  erence . 

1.  I  prefer  'UNLOAD.' 

2.  Neither  Recover  nor  Stall.  You  are  bleeding  energy 
at  high  speed,  so  if  anything,  it  should  be  "Energy." 

3.  I  prefer  "Alpha." 

4.  Would  prefer  both:  Stal l--Recover . 

PHASE  II  --  SIMULATOR 

A)  The  following  comments  are  relevant  to  the  GCAS  warning. 

1.  "CLIMB"  tells  exactly  what  to  do,  and  is  more 
immediate . 

r 

2.  "CLIMB"  is  more  concise  than  "PULL-UP." 

3.  "CLIMB"  is  simpler;  "PULL-UP"  implies  an  abrupt  pitch 
change  which  might  place  the  aircraft  outside  flight 
parameters . 

4.  "PULL-UP"  is  same  as  civilian  systems  -  proper  sense 
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of  urgency. 


5.  Prefer  ‘PULL-UP'  because  I  feel  a  pilot  with  alot  on 
his  mind  may  only  key  on  the  word  'UP.'  I  feel  as  I  was 
working  in  the  simulator  -  my  mind  working  -  'UP'  was  the 
word  I  heard.  'CLIMB'  does  not  stress  the  need  for 
immediate  reaction. 

6.  'CLIMB'  was  too  benign;  did  not  grab  my  attention  or 
convey  sense  of  urgency  that  'PULL-UP'  did. 

B)  The  following  comments  are  relevant  to  the  SCP  violation 
warning . 

1.  'TF-Low'  brings  my  attention  to  the  LARA  which  I  feel 
is  most  important.  'ALTITUDE'  made  me  simply  pull  on  the 
stick  much  like  a  "CLIMB'  command. 

2.  'TF-Low'  relates  the  problem  directly  to  the  TFR 
system  as  opposed  to  'ALTITUDE'  which  could  mean  a  number 
of  things  from  altimeter  problems  to  TFR  problems. 

3.  'ALTITUDE'  is  too  ambiguous  and  easily  confused  with 
GCAS.  Another  message  could  be  'VIOLATION.' 

4.  'TF-Low'  immediately  said  to  me  'get  away  from  the 
ground.'  When  I  heard  'ALTITUDE'  I  looked  at  my  altitude 
and  wondered  what  was  the  deal  with  it. 

5.  'ALTITUDE'  usually  associated  with  medium  altitude 
ATC  clearance.  'TF-Low'  associated  with  death. 


C)  The  following  comments  are  relevant  to  the  A3A/3S 
oaut i on . 
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1.  ’ALPHA-SIDE-SLIP”  ia  mora  indicative  of  the  true 
problem. 


2.  Although  ’ALPHA’  ia  more  conciae  than  ’ALPHA-SIDE- 

j  SLIP,’  ’ALPHA-SIDE-SLIP’  relatea  directly  to  an  alpha 

probe  failure  aa  oppoaed  to  alpha  which  could  mean  the 
*  AOA  tape  failed  or  my  angle-of -attack  waa  too  high. 

3.  Of  all  the  conditions  that  may  cause  an  AOA/SS 
caution  to  be  activated,  only  one  (large  initial  sideslip 
or  flight  control  disconnect  with  flaps  up)  reflects 
sideslip  situation.  All  others  are  alpha  related, 
indicating  a  significant  hazard,  and  generally  could  not 
be  pilot  induced  as  a  large  initial  sideslip  might  be. 

My  feeling  is  the  ’ALPHA’  phrase  is  more  acceptable. 

4.  A  toss  up. 

5.  Preferred  single  word  warning.  Here  -  no  preference. 


D)  The  following  comments  are  relevant  to  the  Autopilot 
failure  caution. 

1.  ’AUTOPILOT’  brings  attention  immediately  to  the 
Autopilot . 

^  2.  ’AUTOPILOT’  is  more  concise. 

•\  3.  Single  word,  ’AUTOPILOT,’  reflects  the  situation  well 

and  ia  less  distracting. 

4.  The  longer  (3-5)  syllable  unique  commands  are  better 
and  more  easily  recognized  in  the  presence  of  radio 
chatter  and  crew  communication. 
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5.  Preferred  a  response  telling  me  it  failed;  not  just 
to  check  the  switches. 


E)  The  following  comments  are  relevant  to  the  redundancy 

t 

issue  for  GCAS. 

€ 

1.  I  would  like  to  see  a  separate  light  -  a  warning 
light  much  like  the  Radar  Alt  Low. 

2.  Any  redundant  warning  system  that  keeps  the  pilot 
from  hitting  the  ground  is  very  useful. 

3.  Voice  should  be  more  than  enough. 

4.  GCAS  can  easily  fail.  Radar  Alt  Low  warning  lamp 
needs  to  be  present  as  a  backup  and  should  be  restored  to 
full  size.  It’s  a  simple  system  backup  to  an  important 
warning . 

5.  I  often  notice  lamp  before  voice. 

6.  I  would  like  to  see  a  visual  cue  on  the  HUD  (D 
model).  During  bombing  operations,  you’re  always  staring 
out- the -window. 

7.  Any  heads-down  light  would  be  worthless.  Fly-up 
bars  on  the  HUD  would  be  useful. 

8.  Recommend  to  flash  LARA  (or  CARA)  light  ^ 

simultaneously  due  to  the  importance  of  the  warning. 


’■'U.S.Governm«nt  Printing  Office:  1988  —  648-054/00037  58 


